The embodiments of the invention in which an exclusive property or privilege is 
claimed are defined as follows: 

1. A method for automating the optimization of search results displayed in a 
search Web page, the method comprising: 

collecting data that represents a performance of a search result, the data originating 
from at least one of a plurality of sources of performance data; 

normalizing the collected data in accordance with a relative importance of the source 
of the data; 

comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible cause for an underperforming search result; and 
adjusting an operation of a search engine that produced the search result in 
accordance with the diagnosis to improve the search result performance. 

2. The method of Claim 1, wherein the at least one of the plurality of sources of 
performance data includes one of an implicit performance data, an explicit performance data, 
a human-judged performance data, a relevance verification data, and a sample test data. 

3. The method of Claim 1, wherein normalizing the collected data in accordance 
with a relative importance of the source of the data includes giving greater weight to the data 
from the more important sources and combining the data to reflect the relative importance of 
the source from which the data originated. 

4. The method of Claim 2, wherein comparing the normalized performance data 
to the expected performance data includes determining that the search result is 
underperforming when the normalized performance is below or substantially below the 
expected performance. 

5. The method of Claim 2, wherein the most important source of data is implicit 
performance data, and normalizing the collected data includes giving implicit performance 
data substantially greater weight when combining the data. 
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6. The method of Claim 2, wherein implicit performance data is automatically 
captured when a user interacts with the search result. 

7. The method of Claim 6, wherein implicit performance data includes at least 
one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

8. The method of Claim 7, wherein implicit performance data further identifies 
an operation that the user performed on the result, including at least one of editing, e-mailing, 
printing, bookmarking, and copying. 

9. The method of Claim 6, wherein implicit performance data includes captured 
data that has been aggregated across multiple interactions with the search result. 

10. The method of Claim 6, wherein implicit performance data includes captured 
data that has been aggregated across multiple users interacting with the search result. 

11. The method of Claim 2, wherein explicit performance data includes data 
obtained from user response to a search engine operator-generated inquiry about the search 
result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry. 

12. The method of Claim 2, wherein human-judged performance data includes 
data obtained from a human evaluation of the search result. 

13. The method of Claim 2, wherein relevance verification data includes data 
obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

14. The method of Claim 2, wherein sample test data includes data obtained from 
a test of the search result's relevance performed on a sample of a subset of users. 
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15. The method of Claim 2, wherein diagnosing at least one possible cause for an 
underperforming search result includes considering at least one of whether the search result 
is no longer valid, whether the search result appears in a poor location, whether a search term 
that generated the search result is easily misspelled, whether the search term is too broad to 
generate a meaningful result, and whether a search for the search term should be constrained 
to a specific resource. 

16. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes modifying the search engine's search schema to change the search 
result generated for the search term, including at least one of reranking, removing, and 
replacing the search results. 

1 7. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes modifying the search engine's search schema to augment a 
presentation of the search results generated for the search term, including at least one of 
highlighting, animating, enlarging, and repositioning an appearance of the search result on a 
search results web page. 

1 8. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes increasing the search engine's spellchecker tolerance. 

19. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes prompting the user to one of clarify or narrow the search term with an 
additional user input. 

20. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes temporarily adjusting the operation of the search engine, and 
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determining whether the adjustments have actually improved the search result performance 
before permanently adjusting the operation of the search engine. 

21. The method of Claim 2, wherein adjusting an operation of a search engine that 
produced the search result in accordance with the diagnosis to improve the search result 
performance, includes adjusting the operation of the search engine in real time. 

22. The method of Claim 21, wherein adjusting the operation of the search engine 
in real time includes intercepting the search result generated by the search engine and 
modifying the search result before the search engine displays the search result to the user. 

23. An automated search result optimization system comprising: 

an input data representing a performance of a search result generated by a search 
engine for a search term, the input data originating from a plurality of sources; 

a data collection process to collect the input performance data from the plurality of 
sources; 

a diagnostic process to compare a performance of the search result, as represented by 
the collected performance data, to an expected performance of the search result; and 

an adjustment process to generate an output data representing an action to 
automatically adjust an operation of the search engine whenever the search result 
performance compares unfavorably to the expected performance, wherein the adjusted 
operation of the search engine is to improve the performance of the search result. 

24. The system of Claim 23, wherein the at least one of the plurality of sources of 
input performance data includes one of an implicit performance data, an explicit performance 
data, a human-judged performance data, a relevance verification data, and a sample test data. 

25. The system of Claim 23, wherein the data collection process includes a 
normalizing process to normalize the collected data in accordance with a relative importance 
of the source of the data, and further wherein normalizing includes giving greater weight to 
the data from the more important sources and combining the data to reflect the relative 
importance of the source from which the data originated. 
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26. The system of Claim 25, wherein the diagnostic process compares the 
performance of the search result as represented by the normalized collected performance data 
to an expected performance of the search result; and further, wherein the comparison is 
unfavorable when the performance is lower than or substantially lower than the expected 
performance. 

27. The system of Claim 25, wherein the most important source of data is implicit 
performance data, and normalizing the collected data includes giving implicit performance 
data substantially greater weight when combining the data. 

28. The system of Claim 27, wherein implicit performance data is automatically 
captured when a user interacts with the search result. 

29. The system of Claim 28, wherein implicit performance data includes at least 
one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

30. The system of Claim 29, wherein implicit performance data further includes 
data representing an identification of an operation that the user performed on the result, 
including at least one of editing, e-mailing, printing, bookmarking, and copying. 

3 1 . The system of Claim 28, wherein implicit performance data includes captured 
data that has been aggregated across multiple interactions with the search result. 

32. The system of Claim 28, wherein implicit performance data includes captured 
data that has been aggregated across multiple users interacting with the search result. 

33. The system of Claim 24, wherein explicit performance data includes data 
obtained from user response to a search engine operator-generated user satisfaction survey 
about the search result. 

34. The system of Claim 24, wherein human-judged performance data includes 
data obtained from a human evaluation of the search result. 
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35. The system of Claim 24, wherein relevance verification data includes data 
obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

36. The system of Claim 24, wherein sample test data includes data obtained from 
a test of the search result's relevance performed on a sample of a subset of users. 

37. The system of Claim 24, wherein the diagnostic process further includes 
diagnosing at least one possible reason why the performance of the search result compares 
unfavorably to the expected performance, including considering at least one of whether the 
search result is no longer valid, whether the search result appears in a poor location, whether 
a search term that generated the search result is easily misspelled, whether the search term is 
too broad to generate a meaningful result, and whether a search for the search term should be 
constrained to a specific resource. 

38. The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
generating output data that represents an action to modify the search engine's search schema, 
wherein the modified search schema changes the way the search engine generates a search 
result for a search term, including at least one of reranking, removing, and replacing the 
search result. 

39. The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
generating output data that represents an action to modify the search engine's search schema, 
wherein the modified search schema augments the way the search engine presents the search 
result for a search term, including at least one of highlighting, animating, enlarging, and 
repositioning an appearance of the search result on a search result Web page. 

40. The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
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generating output data that represents an action to increase the search engine's spellchecker 
tolerance. 

41 . The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
generating output data that represents an action to prompt the user to one of clarify or narrow 
the search term with an additional user input. 

42. The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
generating output data that represents an action to temporarily adjust the operation of the 
search engine, and to further determine whether the temporary adjustment has actually 
improved the search result performance before generating an action to permanently adjust the 
operation of the search engine. 

43. The system of Claim 23, wherein the adjustment process to generate an output 
data representing an action to automatically adjust an operation of the search engine, includes 
generating output data that represents an action to adjust the operation of the search engine in 
real time. 

44. The system of Claim 42, wherein the action to adjust the operation of the 
search engine in real time includes an action to intercept the search result generated by the 
search engine and to further modify the search result before the search engine displays the 
search result to the user. 

45. One or more computer- accessible media having instructions for facilitating 
the automated optimization of a search result in a search result user interface, the instructions 
comprising: 

collecting data that represents a performance of a search result from at least one of a 
plurality of sources of performance data; 

normalizing the collected data in accordance with a relative importance of the source 
of the data; 
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comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible reason why the search result performance compares 
unfavorably to the expected performance; and 

adjusting an operation of a search engine that produced the search result in 
accordance with the diagnosis to improve the search result performance. 

46. The computer-accessible media of Claim 45, wherein the at least one of the 
plurality of sources of performance data includes one of an implicit performance data, an 
explicit performance data, a human-judged performance data, a relevance verification data, 
and a sample test data. 

47. The computer-accessible media of Claim 45, wherein the instruction to 
normalize the collected data in accordance with a relative importance of the source of the 
data includes instructions to give greater weight to the data from the more important sources 
and to combine the data to reflect the relative importance of the source from which the data 
originated. 

48. The computer-accessible media of Claim 45, wherein the instruction to 
compare the normalized performance data to the expected performance data includes an 
instruction to determine that the search result is underperforming when the normalized 
performance is below or substantially below the expected performance. 

49. The computer-accessible media of Claim 47, wherein the most important 
source of data is implicit performance data, and the instruction to normalize the collected 
data includes an instruction to give implicit performance data substantially greater weight 
when combining the data. 

50. The computer-accessible media of Claim 46, wherein implicit performance 
data is data that is automatically captured by a search engine when a user interacts with a 
search result. 
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51. The computer-accessible media of Claim 50, wherein implicit performance 
data includes data that represents at least one of whether the user clicked on the result, a 
location of the result when the user clicked the result, and a length of time that the user 
browsed the result. 

52. The computer-accessible media of Claim 50, wherein implicit performance 
data further identifies an operation that the user performed on the result, including at least 
one of editing, e-mailing, printing, bookmarking, and copying. 

53. The computer-accessible media of Claim 50, wherein implicit performance 
data includes captured data that has been aggregated across multiple interactions with the 
search result. 

54. The computer-accessible media of Claim 50, wherein implicit performance 
data includes captured data that has been aggregated across multiple users interacting with 
the search result. 

55. The computer-accessible media of Claim 46, wherein explicit performance 
data includes data obtained from user response to a search engine operator-generated inquiry 
about the search result, wherein the inquiry includes one of an on-line inquiry and a 
telephone inquiry. 

56. The computer-accessible media of Claim 46, wherein human-judged 
performance data includes data obtained from a human evaluation of the search result. 

57. The computer-accessible media of Claim 46, wherein relevance verification 
data includes data obtained from a verification test of the search result's relevance to verify 
whether a search result having a known relevance is still included in the search result 
generated by the search engine. 
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58. The computer-accessible media of Claim 46, wherein sample test data 
includes data obtained from a test of the search result's relevance performed on a sample of a 
subset of users. 

59. The computer-accessible media of Claim 45, wherein the instruction to 
diagnose at least one possible reason why the search result performance compares 
unfavorably to the expected performance includes an instruction to determine at least one of 
whether the search result is no longer valid, whether the search result appears in a poor 
location, whether a search term that generated the search result is easily misspelled, whether 
the search term is too broad to generate a meaningful result, and whether a search for the 
search term should be constrained to a specific resource. 

60. The computer-accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to change the 
search result generated for the search term, wherein to change the search result includes at 
least one of to rerank, remove, and replace the search result. 

61. The computer-accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to augment a 
presentation of the search result generated for the search term, wherein to augment the 
presentation includes at least one of to highlight, animate, enlarge, and reposition a display of 
the search result on a search result Web page. 

62. The computer-accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to increase the search engine's spellchecker tolerance. 

63. The computer- accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
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diagnosis, includes an instruction to prompt the user to one of clarify or narrow the search 
term with an additional user input. 

64. The computer-accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
diagnosis; includes an instruction to temporarily adjust the operation of the search engine, 
and an instruction to further determine whether the adjustments have actually improved the 
search result performance before performing an instruction to permanently adjust the 
operation of the search engine. 

65. The computer-accessible media of Claim 45, wherein the instruction to adjust 
the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to adjust the operation of the search engine in real time. 

66. The computer-accessible media of Claim 65, wherein the instruction to adjust 
the operation of the search engine in real time includes an instruction to intercept the search 
result generated by the search engine and a further instruction to modify the search result 
before the search engine displays the search result to the user. 
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